LOE COE Documentation (2026)

Table of Contents

1. Overview
2. Project Definition
3. System Architecture
4. Databases
5. Application Features
6. Machine Learning Algorithms
7. Data Processing Workflow
8. Environment Setup
9. Repository Structure
10. Technical Implementation
11. Known Issues and Considerations
12. Future Improvements

---

Overview

LOE COE stands for Likelihood of Event and Consequence of Event, a comprehensive machine learning system designed for predicting water main pipe breaks. This project uses survival analysis and various machine learning algorithms to assess the probability of pipe failures and their potential impact on surrounding infrastructure, particularly buildings.

Key Purpose

• Predicting pipe breaks using historical data and infrastructure characteristics
• Analyzing likelihood-of-event (LOE) scenarios for water main failures
• Evaluating consequence-of-event (COE) by identifying affected buildings and infrastructure
• Implementing transfer learning techniques to apply models trained on one utility to another

---

Project Definition

LOE (Likelihood of Event)

Likelihood of Event represents the probability that a pipe break will occur. This is calculated using survival analysis techniques and machine learning models that analyze:

• Historical pipe break data
• Pipe characteristics (age, material, diameter, installation date)
• Environmental factors
• Temporal patterns
• Spatial relationships

The LOE component outputs partial hazards or risk scores for each pipe segment, indicating the relative likelihood of failure.

COE (Consequence of Event)

Consequence of Event represents the impact or severity if a pipe break were to occur. This is assessed by:

• Identifying buildings and infrastructure affected by a specific water main
• Calculating the number of properties that would lose water service
• Evaluating critical infrastructure dependencies
• Assessing potential service disruption impacts

The COE component connects to building databases to determine which properties are served by specific water mains.

---

System Architecture

Three Main Components

1. Original Pipe Breaks Project

The foundational machine learning project for training and testing pipe break prediction models.

• Repository: pipe-breaks
• Purpose: Algorithm development, model training, data processing
• Command: python script_runner.py -h

2. LOE Application (loe-coe-app)

A Streamlit-based web application that provides a user interface for the pipe break prediction system.

• Repository: loe-coe-app_deven-fork
• Purpose: User-friendly interface for LOE/COE analysis
• Command: streamlit run app.py
• Key Features: Model training, prediction, visualization, scenario management

3. Transfer Learning Implementation

Enables models trained on one utility's data to be applied to another utility.

• Repository: pipe-breaks-transfer-learning
• Purpose: Cross-utility model application
• Command: streamlit run app.py

Repository Evolution

---

Databases

The LOE COE system uses two primary SQLite databases:

1. LOE Database (dnv_coe.db)

Note: The file is currently named dnv_coe.db but should be renamed to dnv_loe.db for accuracy.

Purpose: Stores pipe infrastructure data and survival analysis results

Key Tables:

• survival: Processed dataset for survival analysis containing:

  • Pipe characteristics (material, diameter, length, installation date)
  • Break history and time-to-event data
  • Covariates and features for model training
  • Duration calculations for survival analysis

• scenarios: Tracks different analysis scenarios

  • Run ID for each analysis execution
  • Parameters used in each run
  • Status indicators
  • Timestamp information

Database Schema: Screenshot: LOE database schema (image not in repo yet).

2. COE Database (dnv_coe_buildings.db)

Purpose: Stores building and property data for consequence analysis

Key Tables:

• Building locations and identifiers
• Water main service relationships
• Property information
• Spatial relationships between pipes and buildings

Database Schema: Screenshot: COE database schema (image not in repo yet).

Database Connections

The application uses URI-based database connections:

• LOE Database: URI_SQLITE_DB = "data/dnv_coe.db"
• COE Buildings Database: URI_BLGS_DB = "data/dnv_coe_buildings.db"

---

Application Features

LOE COE Streamlit App Functions

1. Create Shell Script (Training on Compute Clusters)

• Trigger: create_shell_script() function
• Purpose: Generate SLURM scripts for training on compute clusters
• Inputs:
  • Nodes, GPUs, tasks
  • Account credentials
  • Mail type and email
  • Algorithm selection
  • Utility selection
• Process: Constructs SLURM directives and Python commands for cluster execution

2. Find Buildings Affected (COE Analysis)

• Trigger: water_main() function → buildings_affected_by_watermain(asset_id)
• Purpose: Identify buildings impacted by a specific water main
• Database: Connects to dnv_coe_buildings.db
• Process:
  • User inputs an Asset ID (water main identifier)
  • SQL queries retrieve buildings served by that water main
  • Results merged and displayed in the interface
  • Shows list of affected properties and infrastructure

3. Database Connection Check (LOE)

• Trigger: check() function
• Purpose: Verify database connectivity
• Database: Connects to dnv_coe.db (LOE database)
• Output: Success or failure message

4. Create and Clean Dataset (LOE)

• Trigger: dataset_for_surv_analysis() function
• Purpose: Prepare data for survival analysis
• Database: Connects to dnv_coe.db
• Process:
  1. Check if survival table exists
  2. Load data from table if available
  3. Clean and process dataset
  4. Handle missing values
  5. Calculate time-to-event
  6. Merge replacement dates from CSV
  7. Calculate duration using study period settings
  8. Write processed data back to survival table
  9. Optionally save to CSV

5. View Model Parameters

• Trigger: view_params() function
• Purpose: Display current model configuration
• Output: Shows predefined model parameters in markdown format

6. Calculate Results (Model Execution)

• Trigger: config(args) function
• Status: Currently not working (requires debugging)
• Purpose: Execute model training/prediction
• Process:
  • Display processing spinner
  • Load configuration file (.toml) for specified utility
  • Execute model calculations

7. View Results (Visualization)

• Trigger: make_scatter(args) function
• Purpose: Generate and display scatter plots of results
• Output: Visualizations of pipe break predictions and partial hazards

8. Scenario Table (Run Management)

• Trigger: scenarios(args) function
• Purpose: Create new analysis scenario entries
• Process:
  1. Retrieve current run ID from database
  2. Initialize to 1 if none exists, otherwise increment max run ID
  3. Insert new scenario entry with:
     • New run ID
     • Status (set to 0)
     • Parameters used

9. View Scenario Table

• Trigger: view_scenarios(args) function
• Purpose: Display recent analysis scenarios
• Process:
  • Connect to SQLite database
  • Query scenarios table
  • Display last 5 entries
  • Show warning if table not found

---

Machine Learning Algorithms

The LOE COE system supports multiple survival analysis and machine learning algorithms:

1. Cox Proportional Hazards (Cox-PH)

• Module: pipe_breaks.algorithms.cox_ph
• Type: Survival regression
• Description: Applies Cox-PH model to break datasets
• Library: lifelines
• Output: Partial hazards for each pipe
• Reference: Lifelines Cox PH Documentation

2. Weibull Proportional Hazards

• Module: pipe_breaks.algorithms.weibull_ph
• Type: Parametric survival model
• Description: Weibull-based survival analysis
• Library: lifelines

3. Random Survival Forest

• Module: pipe_breaks.algorithms.random_survival_forest_GQC
• Type: Ensemble survival model
• Description: Tree-based survival analysis
• Best Practices: Document pending

4. XGBoost

• Module: pipe_breaks.algorithms.xg_boost
• Type: Gradient boosting
• Best Practices:
  • Add W&B (Weights & Biases) for tracking
  • Plot losses with validation set
  • Configure W&B sweep for hyperparameter tuning
  • Use native saving functionality (don't pickle)

5. XGBSE (XGBoost Survival Embeddings)

• Module: pipe_breaks.algorithms.xgbse_gqc
• Type: Gradient boosting for survival analysis
• Status: No longer maintained (no activity since Mar 2022)
• Best Practices: Same as XGBoost

6. LightGBM

• Module: pipe_breaks.algorithms.lightgbm
• Type: Gradient boosting
• Applications: Used for concordance indices across utilities
• Best Practices:
  • Add W&B for tracking
  • Plot losses with validation set
  • Configure W&B sweep for hyperparameter tuning
  • Use native saving functionality

7. Kaplan-Meier

• Module: pipe_breaks.algorithms.kaplanmeier
• Type: Non-parametric survival analysis
• Description: Basic survival curve estimation
• Library: lifelines
• Reference: Lifelines Kaplan-Meier Documentation

8. NMTLR (Neural Multi-Task Logistic Regression)

• Status: No longer maintained (no updates since Apr 2019)
• Note: Need to find alternatives or maintained implementations

---

Data Processing Workflow

Input Data Requirements

The system requires two primary data sources:

1. Breaks Data: Historical record of pipe failures
2. Mains Data: Pipe infrastructure characteristics

Standard Data Format

Both tables should follow standardized format established in "Standard Table Creation" document.

Processing Pipeline

1. Data Loading

• Function: processing.load_dataset()
• Purpose: Import raw data from databases or CSV files

2. Data Cleaning and Processing

• Function: processing.dataset_for_surv_analysis()

• Steps:

  • Remove duplicates
  • Handle missing values
  • Convert data types
  • Standardize formats

• Key Functions:

  • drop_dup_keep_highest(): Remove duplicate mains, keeping highest scores
  • diameter_conversion(): Standardize pipe diameter measurements
  • days_between(): Calculate time differences in days

3. Time-to-Event Calculation

• Function: calculate_days_num()
• Purpose: Calculate duration from installation to break or censoring
• Inputs: Installation date, break date, study period
• Output: Time-to-event value for survival analysis

4. Feature Engineering

Multiple functions for creating additional features:

• dataset_for_break_date_greater_0_cluster(): Spatial clustering analysis
• dataset_for_num_prev_breaks_greater_0(): Historical break counting
• dataset_for_num_prev_breaks_greater_0_cluster(): Clustered break history

5. Survival Data Creation

• Function: process_full_dataset()
• Output: Complete dataset ready for survival analysis with:
  • Time-to-event (duration)
  • Event indicator (break occurred: yes/no)
  • Covariates (features)
  • Pipe identifiers

Train-Test Split Methodology

Critical Consideration: Temporal nature of data

Original Approach (Problematic):

• Random split of dataset
• Issue: Can use future data to predict past events

Recommended Approach:

1. Sort dataset by install_date
2. Take first 70-80% for training
3. Use remaining 20-30% for testing
4. Ensure test period includes breaks
5. Check distribution of breaks vs. non-breaks
6. Apply SMOTE or similar if class imbalance is significant

Imbalance Handling:

• SMOTE: Synthetic Minority Over-sampling Technique
• Alternatives: AdaSyn, Border SMOTE
• Modern Approach: Adjust loss function weights instead

Data Files Generated

From processing step (in order):

1. {utility}_mains_w_survival_data.csv: Mains with survival calculations
2. {utility}_output_dataframe.csv: Full processed dataset
3. {utility}_output_dataframe_filtered.csv: Filtered for analysis

After algorithm execution: 4. {utility}_{algorithm}_partial_hazards.csv: Main ID + partial hazards for all pipes

After comparison: 5. {utility}_{algorithm}_partial_hazards_main_id.csv: Partial hazards for pipes that broke during test period

---

Environment Setup

Python Environments

Environment Name	Python Version	Key Dependencies	Use Case
pipe_breaks	3.8.10	scikit-learn=1.2.2	LOE-COE app, original pipe breaks
pipe-breaks	3.9.16	-	MSI environment
pipe_breaks_transfer_learning	3.8.10	scikit-learn=0.24.2	Transfer learning app

Development Machines

• HP Laptop: Primary development environment
• MSI: Alternative development environment
• Compute Canada: Model training on HPC clusters

Installation Steps

1. Clone Repository

git clone git@github.com:gqc/loe_coe_app.git
cd loe_coe_app

2. Switch to Development Branch

git checkout dev

3. Create Virtual Environment

mkvirtualenv loe-coe-env

4. Install Requirements

Note: There are 2 requirements files

pip install -r requirements.txt
pip install -r reqs.txt

5. Copy Databases

Prerequisite: WSL must have SSH key to MSI server

scp msi:/home/gqc/git/gqc/loe_coe_app_deven_fork/data .

6. Run Application

streamlit run app.py

Environment Compatibility Notes

Critical: Version compatibility is essential

• Python 3.8.10 recommended for stability
• scikit-learn version mismatches can cause:
  • ModuleNotFoundError: No module named 'sklearn.neighbors._dist_metrics'
  • Index out of bounds errors
  • Module not found errors in Streamlit

Resolution:

• Ensure consistent Python versions between training and testing
• Match scikit-learn versions across environments
• Verify VSCode environment selection (bottom right corner)

---

Repository Structure

Main Repositories

Index	Repository	Description	Machine	Environment	Command
1	pipe-breaks	Original pipe breaks project	HP Laptop / MSI	pipe_breaks (3.8.10) / pipe-breaks (3.9.16)	python script_runner.py -h
2	loe-coe-app_deven-fork	Likelihood-of-Event app fork	HP Laptop / MSI	pipe_breaks (3.8.10) / pipe-breaks (3.9.16)	streamlit run app.py
3	pipe-breaks-transfer-learning	Transfer learning implementation	HP Laptop / MSI	pipe_breaks_transfer_learning (3.8.10)	streamlit run app.py

Documentation Locations

Local Documentation

• ./internal/Pipe Breaks/: Main documentation directory
  • readme.md: Project overview
  • training-and-predictions.md: Training documentation
  • 02_loe-coe-update_7_29_2024.md: LOE COE update
  • 01_setup.md: Setup instructions
  • dnv.md: DNV utility specifics
  • task_list.md: Development tasks

Remote Documentation

• GitHub: https://github.com/gqc/pipe_breaks/tree/master/docs
• Sphinx Documentation: loe-coe-app (gh-pages branch)
• Static App: Available via project links

Code Structure

Pipe Breaks Package Structure

pipe_breaks/
├── algorithms/
│   ├── cox_ph.py
│   ├── weibull_ph.py
│   ├── kaplanmeier.py
│   ├── random_survival_forest_GQC.py
│   ├── xg_boost.py
│   ├── xgbse_gqc.py
│   ├── lightgbm.py
│   └── test_processing.py
├── common/
│   ├── buildings_affected.py
│   ├── create_and_clean.py
│   ├── db_connection.py
│   ├── functions.py
│   ├── plotting.py
│   ├── processing.py
│   ├── spacetime_processing.py
│   └── sqlconnection.py
├── script_runner.py
├── settings.py
├── constants_gqc.py
└── [algorithm]_settings.py

---

Technical Implementation

Model Training Workflow

Available Models

Vannary provided pre-trained models for multiple utilities:

• Calgary (CGY)
• Everett (Eve)
• District of North Vancouver (DNV)
• City of Vancouver (COV)

Two Configurations:

1. Base covariates only
2. Base + additional covariates

Location: Google Drive > VS Research > pipe_breaks > saved_models

Training Flow

New Flow (in development):

Key Functions and Modules

Script Runner (script_runner.py)

Central execution script for running algorithms.

Usage:

python script_runner.py -h

Arguments:

• -s, --scripts: Script to run (required)
• -d, --directory: Directory (optional, default=algorithms)
• -u, --utility: Utility name (optional, default=settings.UTILITY)

Functions:

• parse_args(): Parse command line arguments
• main(): Run specified script with parsed config

Processing Module (common/processing.py)

Core data processing functions.

Key Functions:

• dataset_for_surv_analysis(): Create survival analysis dataset
• process_full_dataset(): Complete dataset processing
• all_unbroken_pipes(): Filter for pipes without breaks
• calculate_days_num(): Calculate time durations
• diameter_conversion(): Standardize diameters
• days_between(): Calculate date differences
• load_dataset(): Load data from sources

Buildings Affected Module (common/buildings_affected.py)

COE analysis functions.

Key Functions:

• buildings_affected_by_watermain(asset_id): Find affected buildings
• main(): Execute COE analysis

Database Connection Module (common/db_connection.py)

Database connectivity management.

Key Functions:

• check(): Test database connection

Plotting Module (common/plotting.py)

Visualization functions.

Key Functions:

• make_scatter(args): Create scatter plots
• make_barplot(): Create bar plots
• make_heatmap(): Create heatmap visualizations

Functions Utilities (common/functions.py)

General utility functions.

Key Functions:

• drop_dup_keep_highest(df): Remove duplicates, keep max scores
• get_grid_row_col(gridnum, columns): Calculate grid positions
• print_metrics(y, y_pred): Display performance metrics
• read_data(csv_path, x_cols, y_col): Load data with column selection
• write_csv_df(path, filename, df, index=False): Safe CSV writing with duplicate checking

SQL Connection Module (common/sqlconnection.py)

SQLite connection management.

Key Functions:

• get_connection(path: str): Get cached database connection

Note: Uses caching to reuse connections, handles thread-safety for Streamlit

Configuration Files

• .toml files: Store utility-specific configurations
• settings.py: Global project settings
• [algorithm]_settings.py: Algorithm-specific settings

---

Known Issues and Considerations

Active Issues (as of May 5, 2025)

Core Functionality Issues

1. Algorithm Migration (#1)

   • Status: Open
   • Need to verify and complete migration of remaining algorithms to Streamlit

2. Results Display (#2)

   • Results showing incorrect algorithm outputs

3. Calculate Results Function

   • Currently not working
   • Requires debugging

4. Dependencies Conflicts (#8)

   • Package version conflicts affecting algorithm execution

5. Database Connections (#13)

   • Need proper connection management
   • Prevent rollbacks after app closure

Architecture Issues

6. Directory Structure (#14)

   • Needs reorganization for Sphinx documentation
   • Folders to reorganize: common, algorithms, settings

7. Database ORM (#15)

   • Need to implement SQLAlchemy
   • Current raw SQL needs modernization

8. Output Management (#10)

   • Rework logic for outputs
   • Create run_id-based folders
   • Similar to runs folder structure

Environment Issues

9. Colab Integration (#6)

   • Issues running in Google Colab

10. Database Settings (#7)

    • Configuration management needed

11. Environment Detection (#11)

    • Detect if environment is Compute Canada
    • Implement environment-specific execution paths

Code Quality Issues

12. Code Formatting (#20)

    • Indentation needs change from 4 to 2 spaces
    • Other linting issues

13. Deprecated Algorithms (#23)

    • xgbse: No activity since Mar 2022
    • NMTLR: No updates since Apr 2019
    • Need alternatives

14. Execution Flow Documentation (#12)

    • Need IDEF0 diagram for entire execution loop

15. API Issues (#16)

    • Missing fake_data_generation_settings
    • Missing files/attributes
    • Need requirements.txt

Database Naming Issue

Current: dnv_coe.db Should be: dnv_loe.db (as it stores LOE data, not COE)

Streamlit Multithreading

• Debugging difficult due to multithreading
• Errors may lack detailed descriptions
• Easier to debug in pipe-breaks repository directly

Model Compatibility

• Models trained in mismatched environments cause errors
• Must ensure:
  • Similar training and testing environments
  • Same development and debug environments
  • Matching library versions

Temporal Data Considerations

• Don't use future data to predict past (train-test split issue)
• Must sort by installation date before splitting
• Ensure test period contains breaks

---

Future Improvements

Recommended Actions

1. Documentation

• Complete IDEF0 execution flow diagram
• Expand algorithm best practices documentation
• Create feature/covariate descriptions
• Document all configuration parameters

2. Code Refactoring

• Implement SQLAlchemy ORM
• Reorganize directory structure
• Apply consistent code formatting
• Add comprehensive unit tests
• Improve logging and error handling

3. Algorithm Maintenance

• Replace or update deprecated algorithms (xgbse, NMTLR)
• Standardize algorithm interfaces
• Implement W&B tracking for all algorithms
• Add hyperparameter tuning pipelines

4. Environment Management

• Standardize Python version (recommend 3.8.10)
• Create comprehensive requirements.txt
• Implement environment detection
• Support multiple execution environments

5. Database Improvements

• Rename dnv_coe.db to dnv_loe.db
• Implement proper connection pooling
• Add database migration scripts
• Create backup and recovery procedures

6. UI/UX Enhancements

• Fix "Calculate Results" functionality
• Improve results display accuracy
• Add better visualization options
• Implement run comparison features

7. Output Management

• Implement run_id-based folder structure
• Add output versioning
• Improve scenario tracking
• Create automated report generation

8. Model Improvements

• Implement fine-tuning capabilities
• Add model versioning
• Create model performance tracking
• Support ensemble predictions

9. Reproducibility

• Set random seeds throughout
• Version control datasets
• Track all hyperparameters
• Implement experiment tracking (W&B, MLflow)

10. Testing

• Create unit tests for all modules
• Add integration tests
• Implement continuous integration
• Add data validation tests

---

Appendix

Utilities Analyzed

1. DNV: District of North Vancouver
2. COV: City of Vancouver
3. CGY: Calgary
4. Eve: Everett
5. MASC: (Additional utility)

Performance Metrics

• Concordance Index: Used to evaluate model performance
  • Measures how well predictions rank actual outcomes
  • Also called C-index or Harrell's C-index
  • Higher values indicate better discrimination

Key Contributors

• Vannary: Model training, feature descriptions, flow path analysis
• Deven: App development, QGIS integration, algorithm implementation
• Sudharshan: Neural network models, algorithm development
• Eric: Early algorithm development
• Srujana: Documentation, environment management

External Resources

• Lifelines Library: Survival analysis in Python
• Weights & Biases: Experiment tracking
• scikit-survival: Survival analysis with scikit-learn
• SMOTE: Handling imbalanced datasets

Related Projects

• HydroTrek: Water distribution modeling platform
• CCTV: Sewer inspection analysis
• DeepVibe: Vibration analysis for infrastructure

---

Contact Information

For questions or issues related to the Pipe Breaks LOE COE project, refer to the contacts list in contacts.md or contact the development team.

---

Document Version: 1.0 Last Updated: October 27, 2025 Compiled From: Internal project documentation, code analysis, and system architecture reviews

Note: This documentation is based on the current state of the LOE COE project. As the project evolves, this document should be updated to reflect new features, fixes, and architectural changes.